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ABSTRACT 

Possible methods of improving the effectiveness of 
curriculum materials are listed and analyzed* Three factors are 
identified as heightening concern over effectiveness: the quantity of 
materials available, increasing decentralization of schooling, and 
the accountability movement* Basic issues discussed include 
evaluation of learning materials, methodology to be used in improving 
effectiveness, improving the channels of communication among the 
various groups concerned with the movement, and the development of 
alternative models by vhich learning materials can be improved. 
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M. Frances Klein 

Education in the United States goes through cycles as new perspectives 
are placed on old ideas, and new ideas evolve out of current social pressures 
and concerns. The concern over the effectiveness of instructional materials is 
one such "new" idea which appears to be of increasing concern to many 
educators and to producers and distributors of instructional materials. 
Perhaps the basic concern over the quality of learning materials is not 
new, but at least three factors have contributed to the increasing impetus 
of what is now referred to as a "movement" in education. 

One factor which has contributed to heightening the concern over the 
quality of learning mr.terials is the quantity now available. This quantity 
provides for choices v;hich never before have been available. Kenneth 
Komoski, executive director of EPIS, reports that there are currently 
available more thrm 300,000 instructional items such as games, filmstrips, 
films, and cassettes. Many of these are considered old, but much of this 
growth has occurred'' ov^r the last 15 years • Prior to the 1960s, the standard 
learning rr.torial for all students was the textbook. This textbook was 
usually chosen by people xcith greater authority than the classroom teacher. 
The teacher was given the text and expected to cover it with all, or at 
least most, rtcdentc in the classrooin. The monotony of covering the 
text was occasionally broken up by a film or filmstrip (sometimes related 
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to the content of the text, though not always) or even more rarely, a 
field trip* 

During the 1960s, Increased financial aid from the federal government 
and private foundations, along with ether resources, contributed to the 
increase in the amount of learning materials available. New content began 
to be found In the traditional curricular areas, and new approaches to 
standard content began to appear. Subject areas rarely found before in 
the precollegiate curriculum began to be included. (Goodlad et al., 
1966; Goodlad, 1964.) Also, forms of media other than texts were used 
to present the content to students—films, filmstrips, transparencies, 
records, realia, cassettes, slides, and study prints were used. No longer 
was a choice among textbooks the basic curriculum decision, but choices 
among what content was to be included and how the content was to be presented 
had to be made. 

Another factor increasing the current concern over the effectiveness 
of learning materials la the increasing decentralization of schooling. 
State departments of education are no longer the major agencies for the 
selection of learning materials. Increasingly, the local school district, 
the school faculty, and the teacher are having significant power over what 
will be the curriculum of the school and classroom and what materials 
will be used to implement it. The principal and teachers of a school are 
more likely to be concerned with specific matters dealing with how effective 
learning materials will be with clearly identified groups of students. 
Therefore, more questions can be expected to be raised about the materials 
and what the impact will be upon the students. 
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A third factor is the accountability movement in education. This is 
undoubtedly related to the broader social concerns of consumerism and the 
demand by the public that products be held, accountable to the advertisement 
claims made about them, as Shackelford has suggested. 

While Shackelford states that he has doubts about the movement to 
improve the effectiveness of instructional materials, I am basically quite 
supportive of it. There are some concerns and reservations which I 
have, however, and some of these I share along with Shackelford. (Some 
points of agreement and disagreement with Shackelford are made explicit 
later in this paper.) My own concerns and reservations are based on some 
issues which I see within the movement. Some basic issues concern what 
is involved in the evaluation of learning materials, the methodology to 
be used in improving their effectiveness, the reporting and use of data 
gathered about the effectiveness of materials, improving the channels 
of communication among the various groups directly concerned with this 
movement, and the development of alternative models by which learning 
materials can be improved. The following paragraphs enlarge upon these 
issues as I interpret them. 

Learner verification of learning materials as a way to increase their 
effectiveness is becoming almost a battle cry for the movement. Factions 
• appear to be developing both in support of and in opposition to learner 
verification. There is confusion over what is meant by those who use this 
term, and how the process is different—if at all— from the process of 
validation. Sometimes the two terms, verification and validation, are 
used interchangeably. Perhaps the confusion is due in part to the infancy 
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stage of conceptualization and methodology and the unsureness of some 
Involved in the movement of what can reasonably be done and what is 
unreasonable to demand given the techniques, knowledge, and skills we 
have available* Regardless of how verification and validation evolve in 
their definitions, or collapse into one as the movement develops, they 
both seem to have at least one common characteristic: one way of improving 
learning materials is to put the materials in the hands of students. No 
matter what label is given to this procedure, it is without question one 
basic way in which to improve the effectiveness of learning materials* 
It is not, however, the only way by which curriculum and instructional 
materials may be improved* 

There are a variety of other procedures involved in improving the 
effectiveness of learning materials. Analysis of the content included 
In the program should be conducted: Is the content accurate; factual; 
free from bias and stereotypes in terms of race, ethnicity, sex, and 
religion; and is the content relevant to the student's future and present 
needs? Curriculum analysis should be done: Do the materials include 
objectives, learning activities, and evaluation devices; are the learning 
activities and evaluation procedures appropriate to the objectives; and 
are the materials suitable for the maturity of the students? An analysis 
of the instructional plan of the materials should also be conducted. 
Attention should be paid to considerations such as what instructional 
methodologies are utilized; whether the teaching methods are appropriate 
to help the students achieve the objectives; what theory of learning is 
utilized; what the role of the teacher is; and what other additional resources 
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are needed to ^ninlen^^pt: the ipaterials-. 

Product developnent Is another area of study which can contribute 
to the Improvement of learning materials. (See, for example, the report 
by Eva Baker and Merlin Wlttrock, Project Guidebook; The Practlcum In 
Instructional Development J Some major topics to be considered In this 
area^ are preparing specifications for a product, designing a protopype, 
and developing the product. 

Shackelford mentions other criteria such as practical durability and 
being reflective of the objectives of the school systems. These I believe 
are also of significance In Improving learning materials, but they are 
not commonly included under the learner verification label. These types 
of analysis, when added to the study of what happens to the materials 
and the students when they use the materials under specified conditions, 
could all make significant contributions to improving the effectiveness 
of learning materials;^ 

A major issue within the field has to do with the methodology utilized 
in improving learning materials. Much of the work in the 1960s utilized the 
traditional control-experimental group design in studying the effects 
of materials. Some present work tends to follow similar designs. This 
is probably an appropriate strategy for studying some problems, but it 
is also likely to be inappropriate for studying other problems^ Skill 
Involving a high degree of spphistication in accurately and comprehensively 
describing such factors as the antecedent conditions to learning and the 
context variables in operation while implementing the materials may be 
an equally valuable strategy. A variety of antecedent conditions— such 
as home and community life of the child, and context variables such as 
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attitudes and skills of teachers and support of the administration and 
parents--may significantly affect learning in different ways using the 
same set of materials. It may be impossible to adequately control enough 
of the significant variables Involved in learning so that materials can 
become the single variable to study* This would require us to search for 
new ways to study and Improve learning materials* 

Much of the data being collected today is centered upon cognitive 
concerns. Cognitive growth has been a long-term concern of schooling, 
and much research has been conducted on it. It is not, however, the only 
concern of schools. Materials on the market claim to have Impact upon 
the affective growth of students as well» The methodology utilized, for 
example, in learner verification, may differ quite significantly as to 
whether cognitive skills, psychomotor skills, affective growth, or a 
combination of these is the major focus of the materials. Cognitive 
skills may more easily lend themselves to paper and pencil Instruments 
yielding quantitative data, while affective concerns such as values, interests, 
and feelings may be studied more effectively through skilled observation. 
This would require that work done in learner verification employ an array 
of different types of data collection and not rely exclusively upon 
one type which may be better suited to other concerns. Control of variables 
where possible, adequate description of others, and continual exploration 
of new methodology for research on significant variables will be needed 
In the future. 

Along with developments in methodology goes the need for careful 
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consideration of how' data--lncludlng data regarding learner verification-- 
are to be reported and used as evidence of the effectiveness of materials* 
The reporting of the data collected during developmental processes Is 
as Important to the movement as the collection of It* It appears that some 
publishers, In particular some of the members of the Educational Media 
Producers Council, have been and are continuing to try out their materials 
with students as they are being developed* Perhaps the concern now should 
be shifted to enlarging and Improving the procedures being used* Efforts 
must also be devoted to studying how best to communicate the results of 
the efforts In a meaningful and accurate way to a variety of Interested 
groups* 

There Is some concern being expressed over how the data collected 
and reported will be used* The publisher who can claim that his materials 
were verified on the largest number of students still may not have the 
best materials, but he may have the most glittering advertisement* To 
equate a large sample with a better product Is not necessarily an accurate 
assumption* 

The problem of generallzablllty Is often raised In relation to data 
which the producer Is expected to collect and report* With the complexities 
Involved In the processes of learning, teaching, and the conduct of 
^schooling, generallzablllty may not prove fco be a very useful term* It 
may not be a realistic expectation to ask producers to collect and provide 
evidence on their materials which allow for a high predictive validity* 
It seems unreasonable to expect that a set of learning materials will have 
the same predictable results when used in an inner-city school and a 
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rural school; with a high socio-economic group in the suburbs of Los 
Angeles and in a coal mining region of West Virginia; or with a racially 
mixed group of 30 students and a group of five white students. Perhaps 
at this point in our development, a more reasonable expectation is that 
the publisher describe as completely as possible the samples of students 
utilized (using characteristics such as maturity level, sex and ethnic 
makeup of groups; and socio-economic status), the conditions under which 
the materials were used, the resulting effects of the materials upon the 
students, and the subsequent improvements made in the materials. This 
is probably a more realistic expectation for publishers than to ask them 
to provide statistical data which might be misinterpreted and misused. 

The availability of such data will not, however, make the decision 
of whether the materials should be used in a given school district or 
classroom. A meaningful and comprehensive report from producers will give 
valuable data to persons who must make choices from among all of the 
learning materials now available. Using this data, a match must be made 
among the materials, the teacher who is to implement them, and the student 
who is to learn from them. The final decision regarding materials to be 
used should be a local decision and one which is thoughtfully and 
systematically made using a variety of data. 

Another issue concerns the channels of communication among the various 
groups involved in improving the effectiveness of learning materials. We 
should avoid a "we-versus-they'* stance among the researchers, producers, 
purchasers, and users of learning materials . The effectiveness of learning 
materials should be viewed as a joint concern, with each group having some 
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•peclal knowledge and skills to contribute • Educators must take the time 
to study publisher's report^ and guides, which should be Included with 
materials* Channels of communication which cut across professional 
responsibility must be found so that producers » recearchers, and practitioners 
may exchange significant data, skills, and knowledge about Improving learning 
materials* 

Finally, a variety of models must be Investigated before the people 
Involved In the movement to Improve the effectiveness of learning materials 
accept one model to guide their activities. One model may never be adequate 
unless It Is formulated as a broad synthesis of several varying models 
developed for different purposes and for different types of materials. 
I*ubllshers must determine what they can realistically do, given their 
constraints and flexibilities* Other agencies may also make contributions. 
For example, EPIE, National Evaluation Systems, and the Social Science 
Education Consortium provide valuable data to their Subscribing members and 
clients. 

The staff of the Curriculum Inquiry Center in the Graduate School 
of Education at U^C.L.A. has proposed that it might function as a type 
of underwriter's laboratory for curriculum and instructional materials. 
It would not be based on the model which Shackelford cites, and apparently 
fears. I completely agree with him in that the process of unden^riting 
learning materials must be different from that of underwriting a toaster 
or automobile. It may be, hov;ever, that certain kinds of data useful 
in decision-making processes can best be supplied by independent agencies 
rather than by the producers who hold a basic vested interest in the 
product • 
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During the months and years ahead, we should be grappling with the 
Issues I hax^e discussed In meaningful, prod.-.ctlvc , and creative ways as 
the field of evaluation of learning materials develops. Guiding all 
these activities should be an awareness of a constant and pervasive challenge. 
That challenge Is to make available to the students effective learning 
materials which will assist in the achievement of educational goals and 
objectives. 

For example, many educators are emphasizing the processes of inquiring, 
valuing, and learning how to learn as aims of education. Individualization 
has long been a means by which educational aims are better achieved. As 
individualization spreads to other areas of activity-ouch as evaluation 
and selection of objectives and learning activlties-.i new charge to 
personalize learning is heard. This will require a fehlft in some of the 
assumptions which appear to be basic to the production of many materials. 
Most materials on the market today appear to be developed with the assumption 
that they will be used with a class of 30. Perhaps an assumption basic to 
nf*r materials should be that the materials will more often be used by 
one student and small groups of students. The kinds of goals and the 
means of achieving them which are being reflected by leaders In the • 
educational community and in the community at large should function as 
directives and challenges to producers of learning materials. 

In conclusion, I am strongly supportive of the movement to improve 
the effectiveness of learning materials. I do hold some basic concerns 
over how the movement is developing now and will develop in the future. 
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Unless careful consideration is given as to how the issues fre analyze 

and directed, the promise some of us see in the moveTnent may go unrealljsed. 

The choices in curriculum wade available by the materials are stimulating 
and full of promise. I value options being available in curriculum, and 
would want to maintain many choices. It would be very unfortunate if 
rigid and unrealistic requirements on producers drastically cut down the 
choices in learning materials. The Imposition of one arbitrary model at 
the state level, or regulations at the district level, for collecting 
and reporting data about the effectiveness of learning materials which allows 
for limited types of evidence could be extremely harmful if it does not 
allow for the kinds of flexibility suggested in this paper. If this 
happens, we may find ourselves with only materials which can teach the 
minutiae of schooling with other significant concerns being overlooked. 
This situation simply must be avoided. It rocvy be avoided, however, 
only if we carefully analyze and evaluate what we are doing to improve 
learning materials and what effects our ac::ivlties are having. 

The development of a new area of study in improving learning materials 
should be viewed as a challenge and an educative process for those involved. 
This process should be carefully guided by* the sharing of knowledge and 
skills by all those involved in and affected by it. 
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